Web Survey Bibliography
Relevance & Research Question: A matrix question format (grid) has may advantages in traditional paper-and-pen surveys design: more questions can be presented in a smaller space, thereby saving paper, printing and mailing costs. With the onset of computer-assisted questionnaires and Internet research, these cost saving arguments are no longer valid. Still, matrix questions remain a widely used tool also in Web surveys (e.g. Couper, 2008). There is some empirical evidence (e.g. Toepoel et al 2005; Petchev, 2006; Callegaro, 2010) that matrix questions lead to more statisficing and more dropout. However, presenting questions as single items on the screen leads to longer response times and may increase response burden. In this study we therefore evaluate an alternative to traditional matrix and scrolling formats for web surveys: the horizontal scrolling matrix (HSM).
Methods & Data: In the HSM questions are presented one-by-one on the screen, while ease of answering is improved (e.g., after an answer is selected, the next question is quickly and automatically presented, there is no need for the next or previous button, etc.), and respondents still have an overview of the number of questions in the matrix through a visual bar. In a randomized field experiment we tested whether the HSM question format does improve data quality over the traditional matrix format. We used a 2 by 3 factorial design: the experimental factors were (1) question format: traditional matrix vs HSM and (2) length of ‘matrix’: 5, 10, and 15 items per matrix.
Results: The results were positive: the HSM-format led to fewer break-offs, better data quality and less satisficing as indicated by well-known response styles, such as non-differentiation and extremeness. The HSM helped improving data quality in the longer matrix conditions, which typically suffer from worse data quality. Finally, respondents evaluated the HSM format very positive.
Added Value: Online researchers can profit from the flexibility of web surveys by moving away from grid formats, which are a heritage of paper and pencil surveys. The HSM is a useful alternative, which prevents respondent demotivation by offering superior usability.
GOR Homepage (abstract) / (full text)
Web survey bibliography - Hox, J. (23)
- Mixed Mode Research: Issues in Design and Analysis; 2016; Hox, J.; De Leeuw, E. D.; Klausch, L. T.
- Internet Panels, Professional Respondents, and Data Quality; 2015; Matthijsse, S.; De Leeuw, E. D.; Hox, J.
- Selection error in single- and mixed mode surveys of the Dutch general population; 2015; Hox, J., Klausch, L. T., Schouten, B.
- Evaluating mixed-mode redesign strategies against benchmark surveys: the case of the Crime Victimization...; 2014; Klausch, L. T., Hox, J., Schouten, B.
- The use of within-subject experiments for estimating measurement effects in mixed-mode surveys ; 2014; Klausch, L. T., Schouten, B., Hox, J.
- Pret met panels [Fun online]; 2013; Roberts, A., de Leeuw, E. D., Hox, J., Klausch, L. T., de Jongh, A.
- Leuker kunnen wij het wel maken. Online vragenlijst design: standaard matrix of scrollmatrix (We can...; 2013; Roberts, A., de Leeuw, E. D., Hox, J., Klausch, L. T., de Jongh, A.
- Internet Coverage and Coverage Bias in Europe: Developments Across Countries and Over Time; 2013; Mohorko, A., de Leeuw, E. D.,Hox, J.
- Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions; 2013; Klausch, L. T., Hox, J., Hox, J., Schouten, B.
- Random versus Systematic Error in a Mixed Mode Online-Telephone Survey; 2013; Hox, J., Scherpenzeel, A., Boeve, A., Boeve, A., de Leeuw, E. D.
- Estimating Measurement Effects of Survey Modes From Between and Within Subject Designs; 2013; Klausch, L. T., Hox, J., Schouten, B.
- Does one really know?: Avoiding noninformative answers in a reliable way.; 2013; de Leeuw, E. D., Boevee, A., Hox, J.
- Design of Web Questionnaires: Matrix Questions or Single Question Formats ; 2012; de Leeuw, E. D., Hox, J., Klausch, L. T., Roberts, A., de Jongh, A.
- Question or Mode Effects in Mixed-Mode surveys: A Cross-cultural study in the Netherlands, Germany,...; 2012; de Leeuw, E. D., Nicolaas, G., Campanelli, P., Hox, J.
- Assessing Measurement Equivalence and Bias of Questions in Mixed-mode Surveys Under Controlled Sample...; 2012; Klausch, L. T., Hox, J., Schouten, B.
- The Representativity of Web Surveys of the General Population compared to Traditional Modes and Mixed...; 2012; Klausch, L. T., Schouten, B., Hox, J.
- Matrix vs. Single Question Formats in Web Surveys: Results from a large scale experiment; 2012; Klausch, L. T., de Leeuw, E. D., Hox, J., de Jongh, A., Roberts , A.
- Flexibility of Web Surveys: Probing 'do-not-know' over the Phone and on the Web; 2011; Hox, J., de Leeuw, E. D.
- Mode Effect or Question Wording? Measurement Error in Mixed Mode Surveys; 2011; de Leeuw, E. D., Hox, J., Scherpenzeel, A.
- Measurement Error in Mixed Mode Surveys: Mode or Question Format?; 2011; de Leeuw, E. D., Hox, J.
- Missing data; 2008; de Leeuw, E. D., Hox, J.
- The influence of advance letters on response in telephone surveys; 2007; de Leeuw, E. D., Callegaro, M., Hox, J., Korendijk, E., Lensvelt-Mulders, G. J.
- The effect of computer-assisted interviewing on data quality: A review.; 1995; de Leeuw, E. D., Hox, J., Snijkers, G.